Insights into Cross-validation

Cross-validation is one of the most widely used techniques, in estimating the Generalization Error of classification algorithms. Though several empirical studies have been conducted, to study the behavior of this method in the past, none of them clearly elucidate the reasons behind the observed behavior. In this paper we study the behavior of the moments (i.e. expected value and variance) of the cross-validation Error and explain the observed behavior in detail. In particular, we provide interesting insights into the behavior of covariance between the individuals runs of cross-validation, which has significant effects on the overall variance. We study this behavior on three classification models which are a mix of parametric and non-parametric models namely, Naive Bayes Classifier – parametric model, Decision Trees and K-Nearest Neighbor Classifier – non-parametric models. The moments are computed using closed form expressions rather than directly using Monte Carlo since the former is shown to be a more viable alternative. The work in this paper complements the prior experimental work done in studying this method since it explains in detail the reasons for the observed trends in the experiments, as opposed to simply reporting the observed behavior.